Toolbox
Tidy Forecasting Workflow
Data preparation (tidy)
Plot the data (visualize)
Define a model (specify)
Train the model (estimate)
Check the model (evaluate)
Produce forecasts (forecast)
library(fpp3)
Model GDP per capita over time
Prepare the relevant variable
gdppc <- global_economy |>
mutate(GDP_per_capita = GDP / Population)
Plot the data (look at one country)
gdppc |>
filter(Country == "Sweden") |>
autoplot(GDP_per_capita) +
labs(y = "$US", title = "GDP per capita for Sweden")

Define the model. In this example, use a trend model from TSLM
TSLM(GDP_per_capita ~ trend()) -> trend_model
Train the model
fit <- gdppc |>
model(trend_model = trend_model)
Warning: 7 errors (1 unique) encountered for trend_model
[7] 0 (non-NA) cases
Check model performance
Produce forecasts
fit |> forecast(h = "3 years")
.mean contains the point forecast, GDP_per_capita contains the
distribution.
fit |>
forecast(h = "3 years") |>
filter(Country == "Sweden") |>
autoplot(gdppc) +
labs(y = "$US", title = "GDP per capita for Sweden")

Some simple methods
Mean method: future values equal to historical mean
Naive method: forecast equal to last observation
Seaonal naive method: forecast equal to last observed value from
same season
Drift method: Naive but forecasts can increase or decrease over
time
Example: beer production
train <- aus_production |>
filter_index("1992 Q1" ~ "2006 Q4")
beer_fit <- train |>
model(
Mean = MEAN(Beer),
`Naive` = NAIVE(Beer),
`Seasonal naive` = SNAIVE(Beer)
)
beer_fc <- beer_fit |> forecast(h = 14)
beer_fc |>
autoplot(train, level = NULL) +
autolayer(
filter_index(aus_production, "2007 Q1" ~ .),
colour = 'black'
) +
labs(
y = "Megalitres",
title = "Forecasts for quarterly beer production"
) +
guides(colour = guide_legend(title = "Forecast"))
Plot variable not specified, automatically selected `.vars = Beer`

Example Google stock price
# Re-index based on trading days
google_stock <- gafa_stock |>
filter(Symbol == "GOOG", year(Date) >= 2015) |>
mutate(day = row_number()) |>
update_tsibble(index = day, regular = TRUE)
# Filter the year of interest
google_2015 <- google_stock |> filter(year(Date) == 2015)
# Fit the models
google_fit <- google_2015 |>
model(
Mean = MEAN(Close),
`Naïve` = NAIVE(Close),
Drift = NAIVE(Close ~ drift())
)
# Produce forecasts for the trading days in January 2016
google_jan_2016 <- google_stock |>
filter(yearmonth(Date) == yearmonth("2016 Jan"))
google_fc <- google_fit |>
forecast(new_data = google_jan_2016)
# Plot the forecasts
google_fc |>
autoplot(google_2015, level = NULL) +
autolayer(google_jan_2016, Close, colour = "black") +
labs(y = "$US",
title = "Google daily closing stock prices",
subtitle = "(Jan 2015 - Jan 2016)") +
guides(colour = guide_legend(title = "Forecast"))

Fitted values and residuals
augment(beer_fit)
There are three new columns added to the original data:
.fitted contains the fitted values;
.resid contains the residuals;
.innov contains the “innovation residuals” which, in
this case, are identical to the regular residuals
Residual diagnostics
A good forecast method will have these properties
- The innovation residuals are uncorrelated. If there are correlations
between innovation residuals, then there is information left in the
residuals which should be used in computing forecasts.
- The innovation residuals have zero mean. If they have a mean other
than zero, then the forecasts are biased.
- The innovation residuals have constant variance. This is known as
“homoscedasticity”.
- The innovation residuals are normally distributed.
Not all are necessary, and models satisfying these may still be able
to be improved.
autoplot(google_2015, Close) +
labs(y = "$US",
title = "Google daily closing stock prices in 2015")

aug <- google_2015 |>
model(NAIVE(Close)) |>
augment()
autoplot(aug, .innov) +
labs(y = "$US",
title = "Residuals from the naïve method")

aug |>
ggplot(aes(x = .innov)) +
geom_histogram() +
labs(title = "Histogram of residuals")

aug |>
ACF(.innov) |>
autoplot() +
labs(title = "Residuals from the naïve method")

google_2015 |>
model(NAIVE(Close)) |>
gg_tsresiduals()

Portmanteau tests for autocorrelation
Box-Pierce test
\[
Q = T \sum_{k=1}^\ell r_k^2,
\]
Ljung-Box test
\[
Q^* = T(T+2) \sum_{k=1}^\ell (T-k)^{-1}r_k^2.
\]
Large values suggest that autocorrelations do not come from a white
noise series.
If the autocorrelations did come from a white noise series, then both
Q and Q∗ would have a χ2 distribution with ℓ degrees of freedom.
aug |> features(.innov, box_pierce, lag=10)
aug |> features(.innov, ljung_box, lag=10)
An alternative simple approach that may be appropriate for
forecasting the Google daily closing stock price is the drift method.
The tidy() function shows the one estimated parameter, the
drift coefficient, measuring the average daily change observed in the
historical data
fit <- google_2015 |> model(RW(Close ~ drift()))
tidy(fit)
augment(fit) |> features(.innov, ljung_box, lag=10)
As with the naïve method, the residuals from the drift method are
indistinguishable from a white noise series.
Distributional forecasts and prediction intervals
Forecast distributions
The point forecast is the mean of the distribution. The distribution
is expected to be normal.
Prediction intervals
| 50 |
0.67 |
| 55 |
0.76 |
| 60 |
0.84 |
| 65 |
0.93 |
| 70 |
1.04 |
| 75 |
1.15 |
| 80 |
1.28 |
| 85 |
1.44 |
| 90 |
1.64 |
| 95 |
1.96 |
| 96 |
2.05 |
| 97 |
2.17 |
| 98 |
2.33 |
| 99 |
2.58 |
Benchmark methods
| Mean |
\(\hat\sigma_h = \hat\sigma\sqrt{1 +
1/T}\) |
| Naive |
\(\hat\sigma_h =
\hat\sigma\sqrt{h}\) |
| Seasonal naive |
\(\hat\sigma_h =
\hat\sigma\sqrt{k+1}\) |
| Drift |
\(\hat\sigma_h =
\hat\sigma\sqrt{h(1+h/(T-1))}\) |
google_2015 |>
model(NAIVE(Close)) |>
forecast(h = 10) |>
hilo()
The hilo() function converts the forecast distributions
into intervals. By default, 80% and 95% prediction intervals are
returned, although other options are possible via the level
argument.
google_2015 |>
model(NAIVE(Close)) |>
forecast(h = 10) |>
autoplot(google_2015) +
labs(title="Google daily closing stock price", y="$US" )

Prediction intervals non-normal distributions
Use bootstrapped residuals.
\[y_t = y_{t-1} + e_t.\] use a
randomly sampled error from the past.
\[y^*_{T+2} = y_{T+1}^* +
e^*_{T+2},\]
where \(e^∗_{T+2}\) is another draw
from the collection of residuals. Continuing in this way, we can
simulate an entire set of future values for our time series.
Doing this repeatedly, we obtain many possible futures. To see some
of them, we can use the generate() function.
fit <- google_2015 |>
model(NAIVE(Close))
sim <- fit |> generate(h = 30, times = 5, bootstrap = TRUE)
sim
google_2015 |>
ggplot(aes(x = day)) +
geom_line(aes(y = Close)) +
geom_line(aes(y = .sim, color = as.factor(.rep)),
data = sim) +
labs(title="Google daily closing stock price", y="$US" ) +
guides(colour = "none")

This is all built into the forecast() function so you do not need to
call generate() directly
fc <- fit |> forecast(h = 30, bootstrap = TRUE)
fc
autoplot(fc, google_2015) +
labs(title = "Google daily closing stock price", y="$US")

google_2015 |>
model(NAIVE(Close)) |>
forecast(h = 10, bootstrap = TRUE, times = 1000) |>
hilo()
Forecasting with decomposition
Forecasting the seasonal component and the seasonally adjusted
component separately.
- Seasonal component usually assumed to be slow or unchanging, so a
seasonal naive method
- Seasonally adjusted component uses any non-seasonal method, eg.
drift, Holt’s, non-seasonal ARIMA
Example US retail employment
us_retail_employment <- us_employment |>
filter(year(Month) >= 1990, Title == "Retail Trade")
dcmp <- us_retail_employment |>
model(STL(Employed ~ trend(window = 7), robust=TRUE)) |>
components() |>
select(-.model)
dcmp |>
model(NAIVE(season_adjust)) |>
forecast() |>
autoplot(dcmp) +
labs(y = "Number of people",
title = "US retail employment")

Figure 5.18 shows naïve forecasts of the seasonally adjusted US
retail employment data. These are then “reseasonalised” by adding in the
seasonal naïve forecasts of the seasonal component
Or, more easily:
fit_dcmp <- us_retail_employment |>
model(stlf = decomposition_model(
STL(Employed ~ trend(window = 7), robust=TRUE),
NAIVE(season_adjust)
))
fit_dcmp |>
forecast() |>
autoplot(us_retail_employment) +
labs(y = "Number of people",
title = "US retail employment")
The ACF of the residuals, shown in Figure 5.20, displays significant
autocorrelations. These are due to the naïve method not capturing the
changing trend in the seasonally adjusted series.
fit_dcmp |> gg_tsresiduals()
Evaluating point forecast accuracy
Functions to subset time series
LS0tCnRpdGxlOiAiQ2ggNSBUUyBUb29sYm94IgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKLS0tCgojIFRvb2xib3gKCiMgVGlkeSBGb3JlY2FzdGluZyBXb3JrZmxvdwoKLSAgIERhdGEgcHJlcGFyYXRpb24gKHRpZHkpCgotICAgUGxvdCB0aGUgZGF0YSAodmlzdWFsaXplKQoKLSAgIERlZmluZSBhIG1vZGVsIChzcGVjaWZ5KQoKLSAgIFRyYWluIHRoZSBtb2RlbCAoZXN0aW1hdGUpCgotICAgQ2hlY2sgdGhlIG1vZGVsIChldmFsdWF0ZSkKCi0gICBQcm9kdWNlIGZvcmVjYXN0cyAoZm9yZWNhc3QpCgpgYGB7cn0KbGlicmFyeShmcHAzKQpgYGAKCiMjIE1vZGVsIEdEUCBwZXIgY2FwaXRhIG92ZXIgdGltZQoKIyMjIFByZXBhcmUgdGhlIHJlbGV2YW50IHZhcmlhYmxlCgpgYGB7cn0KZ2RwcGMgPC0gZ2xvYmFsX2Vjb25vbXkgfD4KICBtdXRhdGUoR0RQX3Blcl9jYXBpdGEgPSBHRFAgLyBQb3B1bGF0aW9uKQpgYGAKCiMjIyBQbG90IHRoZSBkYXRhIChsb29rIGF0IG9uZSBjb3VudHJ5KQoKYGBge3J9CmdkcHBjIHw+CiAgZmlsdGVyKENvdW50cnkgPT0gIlN3ZWRlbiIpIHw+CiAgYXV0b3Bsb3QoR0RQX3Blcl9jYXBpdGEpICsKICBsYWJzKHkgPSAiJFVTIiwgdGl0bGUgPSAiR0RQIHBlciBjYXBpdGEgZm9yIFN3ZWRlbiIpCmBgYAoKIyMjIERlZmluZSB0aGUgbW9kZWwuIEluIHRoaXMgZXhhbXBsZSwgdXNlIGEgdHJlbmQgbW9kZWwgZnJvbSBUU0xNCgpgYGB7cn0KVFNMTShHRFBfcGVyX2NhcGl0YSB+IHRyZW5kKCkpIC0+IHRyZW5kX21vZGVsCmBgYAoKIyMjIFRyYWluIHRoZSBtb2RlbAoKYGBge3J9CmZpdCA8LSBnZHBwYyB8PgogIG1vZGVsKHRyZW5kX21vZGVsID0gdHJlbmRfbW9kZWwpCmBgYAoKYGBge3J9CmZpdApgYGAKCiMjIyBDaGVjayBtb2RlbCBwZXJmb3JtYW5jZQoKIyMjIFByb2R1Y2UgZm9yZWNhc3RzCgpgYGB7cn0KZml0IHw+IGZvcmVjYXN0KGggPSAiMyB5ZWFycyIpCmBgYAoKLm1lYW4gY29udGFpbnMgdGhlIHBvaW50IGZvcmVjYXN0LCBHRFBfcGVyX2NhcGl0YSBjb250YWlucyB0aGUgZGlzdHJpYnV0aW9uLgoKYGBge3J9CmZpdCB8PgogIGZvcmVjYXN0KGggPSAiMyB5ZWFycyIpIHw+CiAgZmlsdGVyKENvdW50cnkgPT0gIlN3ZWRlbiIpIHw+CiAgYXV0b3Bsb3QoZ2RwcGMpICsKICBsYWJzKHkgPSAiJFVTIiwgdGl0bGUgPSAiR0RQIHBlciBjYXBpdGEgZm9yIFN3ZWRlbiIpCmBgYAoKIyBTb21lIHNpbXBsZSBtZXRob2RzCgotICAgTWVhbiBtZXRob2Q6IGZ1dHVyZSB2YWx1ZXMgZXF1YWwgdG8gaGlzdG9yaWNhbCBtZWFuCgotICAgTmFpdmUgbWV0aG9kOiBmb3JlY2FzdCBlcXVhbCB0byBsYXN0IG9ic2VydmF0aW9uCgotICAgU2Vhb25hbCBuYWl2ZSBtZXRob2Q6IGZvcmVjYXN0IGVxdWFsIHRvIGxhc3Qgb2JzZXJ2ZWQgdmFsdWUgZnJvbSBzYW1lIHNlYXNvbgoKLSAgIERyaWZ0IG1ldGhvZDogTmFpdmUgYnV0IGZvcmVjYXN0cyBjYW4gaW5jcmVhc2Ugb3IgZGVjcmVhc2Ugb3ZlciB0aW1lCgojIyBFeGFtcGxlOiBiZWVyIHByb2R1Y3Rpb24KCmBgYHtyfQp0cmFpbiA8LSBhdXNfcHJvZHVjdGlvbiB8PgogIGZpbHRlcl9pbmRleCgiMTk5MiBRMSIgfiAiMjAwNiBRNCIpCmJlZXJfZml0IDwtIHRyYWluIHw+CiAgbW9kZWwoCiAgICBNZWFuID0gTUVBTihCZWVyKSwKICAgIGBOYWl2ZWAgPSBOQUlWRShCZWVyKSwKICAgIGBTZWFzb25hbCBuYWl2ZWAgPSBTTkFJVkUoQmVlcikKICApCmJlZXJfZmMgPC0gYmVlcl9maXQgfD4gZm9yZWNhc3QoaCA9IDE0KQpiZWVyX2ZjIHw+CiAgYXV0b3Bsb3QodHJhaW4sIGxldmVsID0gTlVMTCkgKwogIGF1dG9sYXllcigKICAgIGZpbHRlcl9pbmRleChhdXNfcHJvZHVjdGlvbiwgIjIwMDcgUTEiIH4gLiksCiAgICBjb2xvdXIgPSAnYmxhY2snCiAgKSArCiAgbGFicygKICAgIHkgPSAiTWVnYWxpdHJlcyIsCiAgICB0aXRsZSA9ICJGb3JlY2FzdHMgZm9yIHF1YXJ0ZXJseSBiZWVyIHByb2R1Y3Rpb24iCiAgKSArCiAgZ3VpZGVzKGNvbG91ciA9IGd1aWRlX2xlZ2VuZCh0aXRsZSA9ICJGb3JlY2FzdCIpKQpgYGAKCiMjIEV4YW1wbGUgR29vZ2xlIHN0b2NrIHByaWNlCgpgYGB7cn0KIyBSZS1pbmRleCBiYXNlZCBvbiB0cmFkaW5nIGRheXMKZ29vZ2xlX3N0b2NrIDwtIGdhZmFfc3RvY2sgfD4KICBmaWx0ZXIoU3ltYm9sID09ICJHT09HIiwgeWVhcihEYXRlKSA+PSAyMDE1KSB8PgogIG11dGF0ZShkYXkgPSByb3dfbnVtYmVyKCkpIHw+CiAgdXBkYXRlX3RzaWJibGUoaW5kZXggPSBkYXksIHJlZ3VsYXIgPSBUUlVFKQojIEZpbHRlciB0aGUgeWVhciBvZiBpbnRlcmVzdApnb29nbGVfMjAxNSA8LSBnb29nbGVfc3RvY2sgfD4gZmlsdGVyKHllYXIoRGF0ZSkgPT0gMjAxNSkKIyBGaXQgdGhlIG1vZGVscwpnb29nbGVfZml0IDwtIGdvb2dsZV8yMDE1IHw+CiAgbW9kZWwoCiAgICBNZWFuID0gTUVBTihDbG9zZSksCiAgICBgTmHDr3ZlYCA9IE5BSVZFKENsb3NlKSwKICAgIERyaWZ0ID0gTkFJVkUoQ2xvc2UgfiBkcmlmdCgpKQogICkKIyBQcm9kdWNlIGZvcmVjYXN0cyBmb3IgdGhlIHRyYWRpbmcgZGF5cyBpbiBKYW51YXJ5IDIwMTYKZ29vZ2xlX2phbl8yMDE2IDwtIGdvb2dsZV9zdG9jayB8PgogIGZpbHRlcih5ZWFybW9udGgoRGF0ZSkgPT0geWVhcm1vbnRoKCIyMDE2IEphbiIpKQpnb29nbGVfZmMgPC0gZ29vZ2xlX2ZpdCB8PgogIGZvcmVjYXN0KG5ld19kYXRhID0gZ29vZ2xlX2phbl8yMDE2KQojIFBsb3QgdGhlIGZvcmVjYXN0cwpnb29nbGVfZmMgfD4KICBhdXRvcGxvdChnb29nbGVfMjAxNSwgbGV2ZWwgPSBOVUxMKSArCiAgYXV0b2xheWVyKGdvb2dsZV9qYW5fMjAxNiwgQ2xvc2UsIGNvbG91ciA9ICJibGFjayIpICsKICBsYWJzKHkgPSAiJFVTIiwKICAgICAgIHRpdGxlID0gIkdvb2dsZSBkYWlseSBjbG9zaW5nIHN0b2NrIHByaWNlcyIsCiAgICAgICBzdWJ0aXRsZSA9ICIoSmFuIDIwMTUgLSBKYW4gMjAxNikiKSArCiAgZ3VpZGVzKGNvbG91ciA9IGd1aWRlX2xlZ2VuZCh0aXRsZSA9ICJGb3JlY2FzdCIpKQpgYGAKCiMgRml0dGVkIHZhbHVlcyBhbmQgcmVzaWR1YWxzCgpgYGB7cn0KYXVnbWVudChiZWVyX2ZpdCkKYGBgCgpUaGVyZSBhcmUgdGhyZWUgbmV3IGNvbHVtbnMgYWRkZWQgdG8gdGhlIG9yaWdpbmFsIGRhdGE6CgotICAgYC5maXR0ZWRgIGNvbnRhaW5zIHRoZSBmaXR0ZWQgdmFsdWVzOwotICAgYC5yZXNpZGAgY29udGFpbnMgdGhlIHJlc2lkdWFsczsKLSAgIGAuaW5ub3ZgIGNvbnRhaW5zIHRoZSDigJxpbm5vdmF0aW9uIHJlc2lkdWFsc+KAnSB3aGljaCwgaW4gdGhpcyBjYXNlLCBhcmUgaWRlbnRpY2FsIHRvIHRoZSByZWd1bGFyIHJlc2lkdWFscwoKIyBSZXNpZHVhbCBkaWFnbm9zdGljcwoKQSBnb29kIGZvcmVjYXN0IG1ldGhvZCB3aWxsIGhhdmUgdGhlc2UgcHJvcGVydGllcwoKMS4gIFRoZSBpbm5vdmF0aW9uIHJlc2lkdWFscyBhcmUgdW5jb3JyZWxhdGVkLiBJZiB0aGVyZSBhcmUgY29ycmVsYXRpb25zIGJldHdlZW4gaW5ub3ZhdGlvbiByZXNpZHVhbHMsIHRoZW4gdGhlcmUgaXMgaW5mb3JtYXRpb24gbGVmdCBpbiB0aGUgcmVzaWR1YWxzIHdoaWNoIHNob3VsZCBiZSB1c2VkIGluIGNvbXB1dGluZyBmb3JlY2FzdHMuCjIuICBUaGUgaW5ub3ZhdGlvbiByZXNpZHVhbHMgaGF2ZSB6ZXJvIG1lYW4uIElmIHRoZXkgaGF2ZSBhIG1lYW4gb3RoZXIgdGhhbiB6ZXJvLCB0aGVuIHRoZSBmb3JlY2FzdHMgYXJlIGJpYXNlZC4KMy4gIFRoZSBpbm5vdmF0aW9uIHJlc2lkdWFscyBoYXZlIGNvbnN0YW50IHZhcmlhbmNlLiBUaGlzIGlzIGtub3duIGFzIOKAnGhvbW9zY2VkYXN0aWNpdHnigJ0uCjQuICBUaGUgaW5ub3ZhdGlvbiByZXNpZHVhbHMgYXJlIG5vcm1hbGx5IGRpc3RyaWJ1dGVkLgoKTm90IGFsbCBhcmUgbmVjZXNzYXJ5LCBhbmQgbW9kZWxzIHNhdGlzZnlpbmcgdGhlc2UgbWF5IHN0aWxsIGJlIGFibGUgdG8gYmUgaW1wcm92ZWQuCgpgYGB7cn0KYXV0b3Bsb3QoZ29vZ2xlXzIwMTUsIENsb3NlKSArCiAgbGFicyh5ID0gIiRVUyIsCiAgICAgICB0aXRsZSA9ICJHb29nbGUgZGFpbHkgY2xvc2luZyBzdG9jayBwcmljZXMgaW4gMjAxNSIpCmBgYAoKYGBge3J9CmF1ZyA8LSBnb29nbGVfMjAxNSB8PgogIG1vZGVsKE5BSVZFKENsb3NlKSkgfD4KICBhdWdtZW50KCkKYXV0b3Bsb3QoYXVnLCAuaW5ub3YpICsKICBsYWJzKHkgPSAiJFVTIiwKICAgICAgIHRpdGxlID0gIlJlc2lkdWFscyBmcm9tIHRoZSBuYcOvdmUgbWV0aG9kIikKYGBgCgpgYGB7cn0KYXVnIHw+CiAgZ2dwbG90KGFlcyh4ID0gLmlubm92KSkgKwogIGdlb21faGlzdG9ncmFtKCkgKwogIGxhYnModGl0bGUgPSAiSGlzdG9ncmFtIG9mIHJlc2lkdWFscyIpCmBgYAoKYGBge3J9CmF1ZyB8PgogIEFDRiguaW5ub3YpIHw+CiAgYXV0b3Bsb3QoKSArCiAgbGFicyh0aXRsZSA9ICJSZXNpZHVhbHMgZnJvbSB0aGUgbmHDr3ZlIG1ldGhvZCIpCmBgYAoKYGBge3J9Cmdvb2dsZV8yMDE1IHw+CiAgbW9kZWwoTkFJVkUoQ2xvc2UpKSB8PgogIGdnX3RzcmVzaWR1YWxzKCkKYGBgCgojIyBQb3J0bWFudGVhdSB0ZXN0cyBmb3IgYXV0b2NvcnJlbGF0aW9uCgojIyBCb3gtUGllcmNlIHRlc3QKCiQkClEgPSBUIFxzdW1fe2s9MX1eXGVsbCByX2teMiwKJCQKCiMjIExqdW5nLUJveCB0ZXN0CgokJApRXiogPSBUKFQrMikgXHN1bV97az0xfV5cZWxsIChULWspXnstMX1yX2teMi4KJCQKCkxhcmdlIHZhbHVlcyBzdWdnZXN0IHRoYXQgYXV0b2NvcnJlbGF0aW9ucyBkbyBub3QgY29tZSBmcm9tIGEgd2hpdGUgbm9pc2Ugc2VyaWVzLgoKSWYgdGhlIGF1dG9jb3JyZWxhdGlvbnMgZGlkIGNvbWUgZnJvbSBhIHdoaXRlIG5vaXNlIHNlcmllcywgdGhlbiBib3RoIFEgYW5kIFHiiJcgd291bGQgaGF2ZSBhIM+HMiBkaXN0cmlidXRpb24gd2l0aCDihJMgZGVncmVlcyBvZiBmcmVlZG9tLgoKYGBge3J9CmF1ZyB8PiBmZWF0dXJlcyguaW5ub3YsIGJveF9waWVyY2UsIGxhZz0xMCkKYGBgCgpgYGB7cn0KYXVnIHw+IGZlYXR1cmVzKC5pbm5vdiwgbGp1bmdfYm94LCBsYWc9MTApCmBgYAoKQW4gYWx0ZXJuYXRpdmUgc2ltcGxlIGFwcHJvYWNoIHRoYXQgbWF5IGJlIGFwcHJvcHJpYXRlIGZvciBmb3JlY2FzdGluZyB0aGUgR29vZ2xlIGRhaWx5IGNsb3Npbmcgc3RvY2sgcHJpY2UgaXMgdGhlIGRyaWZ0IG1ldGhvZC4gVGhlIGB0aWR5KClgIGZ1bmN0aW9uIHNob3dzIHRoZSBvbmUgZXN0aW1hdGVkIHBhcmFtZXRlciwgdGhlIGRyaWZ0IGNvZWZmaWNpZW50LCBtZWFzdXJpbmcgdGhlIGF2ZXJhZ2UgZGFpbHkgY2hhbmdlIG9ic2VydmVkIGluIHRoZSBoaXN0b3JpY2FsIGRhdGEKCmBgYHtyfQpmaXQgPC0gZ29vZ2xlXzIwMTUgfD4gbW9kZWwoUlcoQ2xvc2UgfiBkcmlmdCgpKSkKdGlkeShmaXQpCmBgYAoKYGBge3J9CmF1Z21lbnQoZml0KSB8PiBmZWF0dXJlcyguaW5ub3YsIGxqdW5nX2JveCwgbGFnPTEwKQpgYGAKCkFzIHdpdGggdGhlIG5hw692ZSBtZXRob2QsIHRoZSByZXNpZHVhbHMgZnJvbSB0aGUgZHJpZnQgbWV0aG9kIGFyZSBpbmRpc3Rpbmd1aXNoYWJsZSBmcm9tIGEgd2hpdGUgbm9pc2Ugc2VyaWVzLgoKIyBEaXN0cmlidXRpb25hbCBmb3JlY2FzdHMgYW5kIHByZWRpY3Rpb24gaW50ZXJ2YWxzCgojIyBGb3JlY2FzdCBkaXN0cmlidXRpb25zCgpUaGUgcG9pbnQgZm9yZWNhc3QgaXMgdGhlIG1lYW4gb2YgdGhlIGRpc3RyaWJ1dGlvbi4gVGhlIGRpc3RyaWJ1dGlvbiBpcyBleHBlY3RlZCB0byBiZSBub3JtYWwuCgojIyBQcmVkaWN0aW9uIGludGVydmFscwoKfCBQZXJjZW50YWdlIHwgTXVsdGlwbGllciB8CnwtLS0tLS0tLS0tLS18LS0tLS0tLS0tLS0tfAp8IDUwICAgICAgICAgfCAwLjY3ICAgICAgIHwKfCA1NSAgICAgICAgIHwgMC43NiAgICAgICB8CnwgNjAgICAgICAgICB8IDAuODQgICAgICAgfAp8IDY1ICAgICAgICAgfCAwLjkzICAgICAgIHwKfCA3MCAgICAgICAgIHwgMS4wNCAgICAgICB8CnwgNzUgICAgICAgICB8IDEuMTUgICAgICAgfAp8IDgwICAgICAgICAgfCAxLjI4ICAgICAgIHwKfCA4NSAgICAgICAgIHwgMS40NCAgICAgICB8CnwgOTAgICAgICAgICB8IDEuNjQgICAgICAgfAp8IDk1ICAgICAgICAgfCAxLjk2ICAgICAgIHwKfCA5NiAgICAgICAgIHwgMi4wNSAgICAgICB8CnwgOTcgICAgICAgICB8IDIuMTcgICAgICAgfAp8IDk4ICAgICAgICAgfCAyLjMzICAgICAgIHwKfCA5OSAgICAgICAgIHwgMi41OCAgICAgICB8CgojIyBCZW5jaG1hcmsgbWV0aG9kcwoKfCBCZW5jaG1hcmsgbWV0aG9kIHwgJGgkLXN0ZXAgZm9yZWNhc3Qgc2QgICAgICAgICAgICAgICAgICAgICAgICAgICB8CnwtLS0tLS0tLS0tLS0tLS0tLS18LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tfAp8IE1lYW4gICAgICAgICAgICAgfCAkXGhhdFxzaWdtYV9oID0gXGhhdFxzaWdtYVxzcXJ0ezEgKyAxL1R9JCAgICAgIHwKfCBOYWl2ZSAgICAgICAgICAgIHwgJFxoYXRcc2lnbWFfaCA9IFxoYXRcc2lnbWFcc3FydHtofSQgICAgICAgICAgICB8CnwgU2Vhc29uYWwgbmFpdmUgICB8ICRcaGF0XHNpZ21hX2ggPSBcaGF0XHNpZ21hXHNxcnR7aysxfSQgICAgICAgICAgfAp8IERyaWZ0ICAgICAgICAgICAgfCAkXGhhdFxzaWdtYV9oID0gXGhhdFxzaWdtYVxzcXJ0e2goMStoLyhULTEpKX0kIHwKCmBgYHtyfQpnb29nbGVfMjAxNSB8PgogIG1vZGVsKE5BSVZFKENsb3NlKSkgfD4KICBmb3JlY2FzdChoID0gMTApIHw+CiAgaGlsbygpCmBgYAoKVGhlIGBoaWxvKClgIGZ1bmN0aW9uIGNvbnZlcnRzIHRoZSBmb3JlY2FzdCBkaXN0cmlidXRpb25zIGludG8gaW50ZXJ2YWxzLiBCeSBkZWZhdWx0LCA4MCUgYW5kIDk1JSBwcmVkaWN0aW9uIGludGVydmFscyBhcmUgcmV0dXJuZWQsIGFsdGhvdWdoIG90aGVyIG9wdGlvbnMgYXJlIHBvc3NpYmxlIHZpYSB0aGUgYGxldmVsYCBhcmd1bWVudC4KCmBgYHtyfQpnb29nbGVfMjAxNSB8PgogIG1vZGVsKE5BSVZFKENsb3NlKSkgfD4KICBmb3JlY2FzdChoID0gMTApIHw+CiAgYXV0b3Bsb3QoZ29vZ2xlXzIwMTUpICsKICBsYWJzKHRpdGxlPSJHb29nbGUgZGFpbHkgY2xvc2luZyBzdG9jayBwcmljZSIsIHk9IiRVUyIgKQpgYGAKCiMjIFByZWRpY3Rpb24gaW50ZXJ2YWxzIG5vbi1ub3JtYWwgZGlzdHJpYnV0aW9ucwoKVXNlIGJvb3RzdHJhcHBlZCByZXNpZHVhbHMuCgokJHlfdCA9IHlfe3QtMX0gKyBlX3QuJCQgdXNlIGEgcmFuZG9tbHkgc2FtcGxlZCBlcnJvciBmcm9tIHRoZSBwYXN0LgoKJCR5Xipfe1QrMn0gPSB5X3tUKzF9XiogKyBlXipfe1QrMn0sJCQKCndoZXJlICRlXuKIl197VCsyfSQgaXMgYW5vdGhlciBkcmF3IGZyb20gdGhlIGNvbGxlY3Rpb24gb2YgcmVzaWR1YWxzLiBDb250aW51aW5nIGluIHRoaXMgd2F5LCB3ZSBjYW4gc2ltdWxhdGUgYW4gZW50aXJlIHNldCBvZiBmdXR1cmUgdmFsdWVzIGZvciBvdXIgdGltZSBzZXJpZXMuCgpEb2luZyB0aGlzIHJlcGVhdGVkbHksIHdlIG9idGFpbiBtYW55IHBvc3NpYmxlIGZ1dHVyZXMuIFRvIHNlZSBzb21lIG9mIHRoZW0sIHdlIGNhbiB1c2UgdGhlIGBnZW5lcmF0ZSgpYCBmdW5jdGlvbi4KCmBgYHtyfQpmaXQgPC0gZ29vZ2xlXzIwMTUgfD4KICBtb2RlbChOQUlWRShDbG9zZSkpCnNpbSA8LSBmaXQgfD4gZ2VuZXJhdGUoaCA9IDMwLCB0aW1lcyA9IDUsIGJvb3RzdHJhcCA9IFRSVUUpCnNpbQpgYGAKCmBgYHtyfQpnb29nbGVfMjAxNSB8PgogIGdncGxvdChhZXMoeCA9IGRheSkpICsKICBnZW9tX2xpbmUoYWVzKHkgPSBDbG9zZSkpICsKICBnZW9tX2xpbmUoYWVzKHkgPSAuc2ltLCBjb2xvciA9IGFzLmZhY3RvcigucmVwKSksCiAgICAgICAgICAgIGRhdGEgPSBzaW0pICsKICBsYWJzKHRpdGxlPSJHb29nbGUgZGFpbHkgY2xvc2luZyBzdG9jayBwcmljZSIsIHk9IiRVUyIgKSArCiAgZ3VpZGVzKGNvbG91ciA9ICJub25lIikKYGBgCgpUaGlzIGlzIGFsbCBidWlsdCBpbnRvIHRoZSBmb3JlY2FzdCgpIGZ1bmN0aW9uIHNvIHlvdSBkbyBub3QgbmVlZCB0byBjYWxsIGdlbmVyYXRlKCkgZGlyZWN0bHkKCmBgYHtyfQpmYyA8LSBmaXQgfD4gZm9yZWNhc3QoaCA9IDMwLCBib290c3RyYXAgPSBUUlVFKQpmYwpgYGAKCmBgYHtyfQphdXRvcGxvdChmYywgZ29vZ2xlXzIwMTUpICsKICBsYWJzKHRpdGxlID0gIkdvb2dsZSBkYWlseSBjbG9zaW5nIHN0b2NrIHByaWNlIiwgeT0iJFVTIikKYGBgCgpgYGB7cn0KZ29vZ2xlXzIwMTUgfD4KICBtb2RlbChOQUlWRShDbG9zZSkpIHw+CiAgZm9yZWNhc3QoaCA9IDEwLCBib290c3RyYXAgPSBUUlVFLCB0aW1lcyA9IDEwMDApIHw+CiAgaGlsbygpCmBgYAoKIyBGb3JlY2FzdGluZyB3aXRoIHRyYW5zZm9ybWF0aW9ucwoKIyMgUHJlZGljdGlvbiBpbnRlcnZhbHMgd2l0aCB0cmFuc2Zvcm1hdGlvbnMKCkluIGdlbmVyYWwsIGJhY2stdHJhbnNmb3JtYXRpb24gaXMgcGVyZm9ybWVkIGF1dG9tYXRpY2FsbHkgYnkgYGZhYmxlYC4KCiMjIEJpYXMgYWRqdXN0bWVudHMKCk1hdGhlbWF0aWNhbCB0cmFuc2Zvcm1hdGlvbnMgbGlrZSBCb3gtQ294IHJldHVybiBtZWRpYW4gaW5zdGVhZCBvZiBtZWFuLiBJbiBvcmRlciB0byB1c2UgdGhlIG1lYW4sIHdlIHVzZSBiaWFzLWFkanVzdGVkIHBvaW50IGZvcmVjYXN0cy4KCmBgYHtyfQpmYyA8LSBwcmljZXMgfD4KICBmaWx0ZXIoIWlzLm5hKGVnZ3MpKSB8PgogIG1vZGVsKFJXKGxvZyhlZ2dzKSB+IGRyaWZ0KCkpKSB8PgogIGZvcmVjYXN0KGggPSA1MCkgfD4KICBtdXRhdGUoLm1lZGlhbiA9IG1lZGlhbihlZ2dzKSkKZmMgfD4KICBhdXRvcGxvdChwcmljZXMgfD4gZmlsdGVyKCFpcy5uYShlZ2dzKSksIGxldmVsID0gODApICsKICBnZW9tX2xpbmUoYWVzKHkgPSAubWVkaWFuKSwgZGF0YSA9IGZjLCBsaW5ldHlwZSA9IDIsIGNvbCA9ICJibHVlIikgKwogIGxhYnModGl0bGUgPSAiQW5udWFsIGVnZyBwcmljZXMiLAogICAgICAgeSA9ICIkVVMgKGluIGNlbnRzIGFkanVzdGVkIGZvciBpbmZsYXRpb24pICIpCmBgYAoKVGhlIGRhc2hlZCBsaW5lIGluIEZpZ3VyZSA1LjE3IHNob3dzIHRoZSBmb3JlY2FzdCBtZWRpYW5zIHdoaWxlIHRoZSBzb2xpZCBsaW5lIHNob3dzIHRoZSBmb3JlY2FzdCBtZWFucy4KCiMgRm9yZWNhc3Rpbmcgd2l0aCBkZWNvbXBvc2l0aW9uCgpGb3JlY2FzdGluZyB0aGUgc2Vhc29uYWwgY29tcG9uZW50IGFuZCB0aGUgc2Vhc29uYWxseSBhZGp1c3RlZCBjb21wb25lbnQgc2VwYXJhdGVseS4KCi0gICBTZWFzb25hbCBjb21wb25lbnQgdXN1YWxseSBhc3N1bWVkIHRvIGJlIHNsb3cgb3IgdW5jaGFuZ2luZywgc28gYSBzZWFzb25hbCBuYWl2ZSBtZXRob2QKLSAgIFNlYXNvbmFsbHkgYWRqdXN0ZWQgY29tcG9uZW50IHVzZXMgYW55IG5vbi1zZWFzb25hbCBtZXRob2QsIGVnLiBkcmlmdCwgSG9sdCdzLCBub24tc2Vhc29uYWwgQVJJTUEKCiMjIEV4YW1wbGUgVVMgcmV0YWlsIGVtcGxveW1lbnQKCmBgYHtyfQp1c19yZXRhaWxfZW1wbG95bWVudCA8LSB1c19lbXBsb3ltZW50IHw+CiAgZmlsdGVyKHllYXIoTW9udGgpID49IDE5OTAsIFRpdGxlID09ICJSZXRhaWwgVHJhZGUiKQpkY21wIDwtIHVzX3JldGFpbF9lbXBsb3ltZW50IHw+CiAgbW9kZWwoU1RMKEVtcGxveWVkIH4gdHJlbmQod2luZG93ID0gNyksIHJvYnVzdD1UUlVFKSkgfD4KICBjb21wb25lbnRzKCkgfD4KICBzZWxlY3QoLS5tb2RlbCkKZGNtcCB8PgogIG1vZGVsKE5BSVZFKHNlYXNvbl9hZGp1c3QpKSB8PgogIGZvcmVjYXN0KCkgfD4KICBhdXRvcGxvdChkY21wKSArCiAgbGFicyh5ID0gIk51bWJlciBvZiBwZW9wbGUiLAogICAgICAgdGl0bGUgPSAiVVMgcmV0YWlsIGVtcGxveW1lbnQiKQpgYGAKCkZpZ3VyZSA1LjE4IHNob3dzIG5hw692ZSBmb3JlY2FzdHMgb2YgdGhlIHNlYXNvbmFsbHkgYWRqdXN0ZWQgVVMgcmV0YWlsIGVtcGxveW1lbnQgZGF0YS4gVGhlc2UgYXJlIHRoZW4g4oCccmVzZWFzb25hbGlzZWTigJ0gYnkgYWRkaW5nIGluIHRoZSBzZWFzb25hbCBuYcOvdmUgZm9yZWNhc3RzIG9mIHRoZSBzZWFzb25hbCBjb21wb25lbnQKCk9yLCBtb3JlIGVhc2lseToKCmBgYHtyfQpmaXRfZGNtcCA8LSB1c19yZXRhaWxfZW1wbG95bWVudCB8PgogIG1vZGVsKHN0bGYgPSBkZWNvbXBvc2l0aW9uX21vZGVsKAogICAgU1RMKEVtcGxveWVkIH4gdHJlbmQod2luZG93ID0gNyksIHJvYnVzdD1UUlVFKSwKICAgIE5BSVZFKHNlYXNvbl9hZGp1c3QpCiAgKSkKZml0X2RjbXAgfD4KICBmb3JlY2FzdCgpIHw+CiAgYXV0b3Bsb3QodXNfcmV0YWlsX2VtcGxveW1lbnQpICsKICBsYWJzKHkgPSAiTnVtYmVyIG9mIHBlb3BsZSIsCiAgICAgICB0aXRsZSA9ICJVUyByZXRhaWwgZW1wbG95bWVudCIpCmBgYAoKVGhlIEFDRiBvZiB0aGUgcmVzaWR1YWxzLCBzaG93biBpbiBGaWd1cmUgNS4yMCwgZGlzcGxheXMgc2lnbmlmaWNhbnQgYXV0b2NvcnJlbGF0aW9ucy4gVGhlc2UgYXJlIGR1ZSB0byB0aGUgbmHDr3ZlIG1ldGhvZCBub3QgY2FwdHVyaW5nIHRoZSBjaGFuZ2luZyB0cmVuZCBpbiB0aGUgc2Vhc29uYWxseSBhZGp1c3RlZCBzZXJpZXMuCgpgYGB7cn0KZml0X2RjbXAgfD4gZ2dfdHNyZXNpZHVhbHMoKQpgYGAKCiMgRXZhbHVhdGluZyBwb2ludCBmb3JlY2FzdCBhY2N1cmFjeQoKIyMgRnVuY3Rpb25zIHRvIHN1YnNldCB0aW1lIHNlcmllcwo=